Comments for MEDB 5501, Module01

Topics to be covered

What you will learn
- About this class
- R and RStudio
- Programming assignments
- Scales of measurement
- Tests of hypothesis
- Confidence intervals
- A simple R program

Welcome to the class

Three sections
- 0001, Synchronous Zoom meetings on Tuesdays
- 0002, Asynchronous
- 0003, International students
Introductions
- Ricardo Moniz
- Suman Sahil

There are three sections in the class.

Section 0001 is a synchronous online class with lectures delivered via Zoom on Tuesdays from 1:00pm to 3:45pm. Students in this section are required to attend that class. If you are in this section, you must show up on Tuesday afternoons. Attendance will be taken. Two or more unexcused absences will require a discussion with the instructor and remedial actions may need to be taken.

Section 0002 is an asynchronous online class where you watch videos of the zoom session. Students in this section have the option of attending the Tuesday 1pm Zoom sessions, if their work schedule permits.

Section 0003 is a synchronous online class like section 0001 with the additional requirement of weekly sign-ins. The weekly sign-in meets an important requirement of some international students. Other than the sign-in, the class is identical to the synchronous online class.

All three sections are combined in Canvas to a single site. This makes it easier for me to make announcements, do grading, etc. Just because the Canvas site says “2024FS-MEDB-5501-0001” does not mean that you are in the wrong Canvas site and it does not mean that you are registered for the wrong class. We’re all one big family from the perspective of Canvas. Only the attendance requirements differ.

Requirements of all students

Attend Tuesday Zoom or watch video of Tuesday Zoom
Read book chapter
Optional review session on Zoom on Fridays
Complete all assignments by Monday at 11:59pm

Attendance

Synchronous students
- MUST attend Tuesday Zoom sessions.
- can also review the recordings
Asynchronous students
- watch the Tuesday recordings
- or attend some of the Tuesday Zoom sessions
- or both
Failure to attend is a problem

Assignments

Due Mondays at 11:59pm
Policy on Failure to submit on time is a problem

This class is in transition

Taught for many years by Dr. Monica Gaddis
My second time teaching this class
Major changes
- Software agnosticism
  - In class switch from SPSS to R
- Suman Sahil will co-teach and cover programming in R
- Discussion boards ask for feedback
- Proposed exam questions

I’m excited at the opportunity to teach this class, but I am also a bit scared. The topics in this class are relatively easy, but teaching them well is not so easy.

This class has been taught for many years by Dr. Monica Gaddis. She is an excellent teacher, but she retired this summer. I have a lot of respect for her and have kept a lot of her material. There are some important differences, however. I am not a wall flower, and I have a few ideas that might supplement an already good class.

I believe in software agnosticism. You should use whatever software you thing is best for you. Use the software that you expect to use in the real world after you graduate. It’s more work for me (and for Suman Sahil, who has to grade the various programs). Nevertheless, I want to make that effort because I should not presume to know what program is best for you and your budding career.

All the classroom examples will use R. If you use another program, expect to spend some time reading the documentation and figuring out how to convert the R code to code for your system. You can ask for assistance by email, during the help session if all the other questions are already answered, or by special appointment. I take great pride in knowing how to do things in a broad range of statistical software and should be able to help with most issues.

If you’ve never used any statistical software before, I would certainly encourage you to use R.

A second change is that I will include a brief feedback assignment after every module. I want to feedback on what you thought was most important, what you found most confusing, and what you might have liked to learn that wasn’t in the lecture.

Don’t spend a lot of time on the feedback. You are welcome to say something brief like nothing was confusing. If everything was confusing, that’s fine also, but please feel free to show up at the Friday review session or set up a special appointment if this is the case.

You are also welcome to say something like “I agree with” and mention the name of a previous student.

I might make some other small changes over time.

There’s a risk in making changes. It may create some inconsistencies between what I am asking you to do and what Dr. Gaddis covered in her videos. Use a bit of common sense, but if you are not quite sure about anything, please ask.

Why R

Faculty consensus
- Prepares you better for capstone/thesis work
- More likely to be used in your future job
- Integrates well with team based tools
R is not as difficult as claimed
- Work from existing templates
Python, SAS, Stata would have also been good choices
- You are welcome to use these if you like

Last year, I taught the class using SPSS and it was an okay choice. But some of the faculty raised concerns about leaving students ill prepared if they needed to do data analysis for a capstone project or thesis. SPSS is okay for introductory statistics, but falls short for the sort of work needed in a thesis.

While some jobs out in the real world will use SPSS, far more will use R.

Most important, in my opinion, is that R integrates well with team-based tools like version control, cloud-based computing, distributed databases, and other methdologies that are being used more and more with programming teams.

You may have heard that R is a difficult language. R is not as difficult as some people may claim. It is being used successfully at a large number of universities in their Statistics and Data Science program. It is even used extensively in undergraduate classes.

Having said that, I have to admit that R is still not as easy to use as SPSS. I will try to lighten that burden by providing templates that will guide your coding throughout all of the assignments.

Now there are several good alternatives to R. Each has its advantages but also some disadvantages. It is not worth quibbling here. I’ve chaired a couple of panel discussion sessions at national Statistics conferences on this topic. My recommendation is to use what your boss uses.

If you want to use something other than R, go for it! You’ll have to do a bit of work on your own, but I will provide support as needed. I take great pride in being able to work with a broad range of statistical software systems.

Student learning objectives, 1 of 3

SLO1
- The graduate will be able to use statistics to analyze and interpret data. They will understand the fundamentals of the field in the context of recognizing the effective use of data or information for the specific discipline(s). They will select and apply appropiate statistical procedures to the information. They will be able to analyze and accurately interpret of statistical result.

Student learning objectives, 2 of 3

SLO2
- The graduate will be able to design a testable research question or hypothesis. They will have adequate background knowledge about biological, biomedical, or population health contexts and problems including common research problems in order to generate a research question or hypothesis. They will be able to relate problems within and across levels of areas of the spectrum to bridge disciplines.

Student learning objectives, 3 of 3

SLO5
- The graduate will be able to communicate scientific outcomes. This includes the ability to convey scientific methods and statistical findings, effectively field questions in an oral presentation format as well as in the preparation of thesis or capstone manuscripts.

Break #1

What you have learned
- About this class
What’s coming next
- R and RStudio

R

RStudio

Other statistical software

JMP, Python, R, SAS, SPSS, Stata
- Use only if you are confident in your abilities
- Avoid Microsoft Excel

Break #2

What you have learned
- R and RStudio
What’s coming next
- Programming assignments

Test

Break #3

What you have learned
- Programming assignments
What’s coming next
- Scales of measurement

Scales of measurement

Dichotomy
- Continuous
- Categorical
Stevens scales of measurement (controversial!)
- Nominal
- Ordinal
- Interval
- Ratio
Addition/subtraction not allowed for ordinal data
- Mean of ordinal data is meaningless

An example of ordinal data.

“Do you agree or disagree with the following statements”
- “I believe that knowledge of Statistics is important for my job.”
  - 1 = Strongly disagree,
  - 2 = Disagree
  - 3 = Neutral
  - 4 = Agree
  - 5 = Strongly agree

Speaker notes

An example of ordinal data is the Likert scale. This takes various forms, but often it is used with group of questions on a questionnaire that reads something like

“Do you agree or disagree with the following statements”

You are asked to respond 1=Strongly disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly agree.

Now I’m sure everyone today is going to choose 5. But assigning numbers 1, 2, 3, 4, and 5 to categories of strongly disagree, disagree, neutral, agree, and strongly agree may falsely imply that a jump from 3 (neutral) to 4 (agree) is about the same amount of improvement as a jump from 4 (agree) to 5 (strongly agree). That’s probably not the case.

You can’t really average ordinal data, some people say because that implies that two responses of “Agree” are the same as one response of “Neutral” along with a response of “Strongly agree”.

Do you want everyone to be at least somewhat on your side or do you want to have a smaller number of very enthusiastic supporters.

If you believe that two 4’s are not the same as a 3 and a 5, then you can’t average.

Now I beg to disagree here, but I am part of a minority opinion. I think that if at the start of this class, your average rating was 3.2 and after I finish the lecture, your average rating climbs to 4.4, that I have done my job well.

If it only jumps to 3.6, then I have still done well, but not as much as that jump to 4.4.

Another example of ordinal data, course grades

A = 4
B = 3
C = 2
D = 1
F = 0

Speaker notes

Another example of ordinal data is grades assigned to students. Now everyone in this class is getting an A, but in other classes I teach I might assign different grades. You can attach a number to each of these grades, 4 for A, 3 for B, 2 for C, 1 for D, and 0 for F.

These numbers seem to imply that a student with two B’s is as smart as a student with an A and a C.

It raises an interesting story. A colleague of mine told me that he would never hire anyone with a single F on their transcript. An F is a red flag, he felt. So he would not want to assign a value of 0 to F, because that implies that the difference between an F and a D is equivalent to the difference between a B and and A. He’s want to assign a value like negative one million to an F so that the average would be pulled way down for a single F, no matter what the other grades would be.

Now I would never be so harsh, but there is really nothing wrong with his perspective. And I would certainly treat a student with three A’s and one F differently from a student with two A’s and two C’s even though mathematically, both average out to 3.0.

Now, in spite of all the obvious problems with equivalence between different grades, most of us still accept a grade point average as a meaningful indicator of how well a student did in school.

Break #4

What you have learned
- Scales of measurement
What’s coming next
- Tests of hypothesis

What is a population?

Population: a group that you wish to generalize your research results to. It is defined in terms of
- Demography,
- Geography,
- Occupation,
- Time,
- Care requirements,
- Diagnosis,
- Or some combination of the above.

Example of a population

All infants born in the state of Missouri during the 1995 calendar year who have one or more visits to the Emergency room during their first year of life.

What is a sample?

Sample: subset of a population.
Random sample: every person has the same probability of being in the sample.
Biased sample: Some people have a decreased probability of being in the sample.
- Always ask “who was left out?”

An example of a biased sample

A researcher wants to characterize illicit drug use in teenagers. She distributes a questionnaire to students attending a local public high school
(in the U.S. high school is grades 9-12, which is mostly students from ages 14 to 18.)
Explain how this sample is biased.
Who has a decreased or even zero probability of being selected.

Type your ideas in the chat box.

Fixing a biased sample

Redfine your population
- Not all teenagers,
  - but those attending public high schools.

What is a parameter?

A parameter is a number computed from a population.
- Examples
  - Average health care cost associated with the 29,637 children
  - Proportion of these 29,637 children who died in their first year of life.
  - Correlation between gestational age and number of ER visits of these 29,637 children.
- Designated by Greek letters (\(\mu\), \(\pi\), \(\rho\))

What is a statistic?

A statistic is a number computed from a sample
- Examples
  - Average health care cost associated with 100 children.
  - Proportion of these 100 children who died in their first year of life.
  - Correlation between gestational age and number of ER visits of these 100 children.
- Designated by non-Greek letters (\(\bar{X}\), \(\hat{p}\), r).

What is Statistics?

Statistics
- The use of information from a sample (a statistic) to make inferences about a population (a parameter)
  - Often a comparison of two populations

What is the null hypothesis?

The null hypothesis (\(H_0\)) is a statement about a parameter.
It implies no difference, no change, or no relationship.
- Examples
  - \(H_0:\ \mu_1 - \mu_2 = 0\)
  - \(H_0:\ \pi_1 - \pi_2 = 0\)
  - \(H_0:\ \rho = 0\)

What is the alternative hypothesis?

The alternative hypothesis (\(H_1\) or \(H_a\)) implies a difference, change, or relationship.
- Examples
  - \(H_1:\ \mu_1 - \mu_2 \ne 0\)
  - \(H_1:\ \pi_1 - \pi_2 \ne 0\)
  - \(H_1:\ \rho \ne 0\)

Hypothesis in English instead of Greek

Only statisticians like Greek letters
- Translate to simple text
- For two group comparisons
  - Safer, more effective
- For regression models
  - Trend, association

Use PICO

P = patient population
I = intervention
C = control
O = outcome

Example of text hypotheses (1/2)

“… the objective of this 78-week randomised, placebo-controlled study was to determine whether treatment with nilvadipine sustained-release 8 mg, once a day, was effective and safe in slowing the rate of cognitive decline in patients with mild to moderate Alzheimer disease.”
- Lawlor B, Segurado R, Kennelly S, et al. Nilvadipine in mild to moderate Alzheimer disease: A randomised controlled trial. PLoS Med. 2018; 15(9): e1002660. DOI: 10.1371/journal.pmed.1002660

PICO for this study

P = patients with mild to moderate Alzheimer disease
I = Nilvadine
C = placebo
O = cognitive function

Example of text hypotheses (2/2)

“… we investigated trends in BCC incidence over a span of 20 years and the associations between incident BCC and risk factors in a total population of 140,171 participants from 2 large US-based cohort studies: women in the Nurses’ Health Study (NHS; 1986–2006) and men in the Health Professionals’ Follow-up Study (HPFS; 1988–2006).”
- Wu S, Han J, Li WQ, Li T, Qureshi AA. Basal-cell carcinoma incidence and associated risk factors in U.S. women and men. Am J Epidemiol. 2013; 178(6): 890–897. DOI: 10.1093/aje/kwt073

PICO for this study

P = female nurses/male health professionals
I = various risk factors
C = absence of various risk factors
O = presence/absence of BCC

One-sided alternatives

Examples
- \(H_1:\ \mu_1 - \mu_2 \gt 0\)
- \(H_1:\ \pi_1 - \pi_2 \gt 0\)
- \(H_1:\ \rho \gt 0\)
Changes in only one direction expected
Changes in opposite direction uninteresting

Passive smoking controversy

EPA meta-analysis of passive smoking
- Criticized for using a one-sided hypothesis
- Samet JM, Burke TA. Turning science into junk: the tobacco industry and passive smoking. Am J Public Health. 2001;91(11):1742–1744.

Speaker notes

Available in html format or PDF format.

Consider a study of the effects of second-hand smoke. These studies always use directional alternatives. From what we know about active cigarette smoking is that it increases the risk of cancer and cardiovascular disease. So there is no reason to expect that passive smoke exposure should be any different than active smoking. Maybe it is less toxic, because of dilution and because the smoking coming off a cigarette from one end is different than the smoke coming off the cigarette from the other end. Fair enough, but there is not reason to believe that things are so different that all of a sudden the smoke becomes protective.

Since there is no scientific basis for a protective effect of passive smoking, it makes sense to test that passive smoking has no effect versus it having an increase in bad outcomes compared to the control group. So your null hypothesis is “not harmful” and your alternative is “harmful”. The beneficial hypothesis is lumped into the null hypothesis, but no one would dare claim that passive smoking was protective.

Actually, the tobacco companies did complain that the use of a directional alternative violated the norms of science. They won in a court battle in North Carolina, but lost on appeal.

As another aside, I was involved with prayer study. We planned this study using a one-sided hypothesis (remote prayer has a positive effect on health). The Institutional Review Board suggested changing this to a two-sided hypothesis (remote prayer has either a positive or a negative effect on health). Thankfully, we did not observe an outcome in the opposite tail as that would have been very difficult to explain.

What is a decision rule? (1/3)

Example
- \(H_0:\ \mu_1 - \mu_2 = 0\)
- \(H_1:\ \mu_1 - \mu_2 \ne 0\)
- t = (\(\bar{X}_1-\bar{X}_2\)) / se
- Accept \(H_0\) if t is close to zero.

What is a decision rule? (2/3)

Example
- \(H_0:\ \pi_1 - \pi_2 = 0\)
- \(H_1:\ \pi_1 - \pi_2 \ne 0\)
- t = (\(\hat{p}_1-\hat{p}_2\)) / se
- Accept \(H_0\) if t is close to zero.

What is a decision rule? (3/3)

Example
- \(H_0:\ \rho = 0\)
- \(H_1:\ \rho \ne 0\)
- t = r / se
- Accept \(H_0\) if t is close to zero.

What is a Type I error?

A Type I error is rejecting the null hypothesis when the null hypothesis is true
- False positive
- Example involving drug approval: a Type I error is allowing an ineffective drug onto the market.
\(\alpha\) = P[Type I error]

Speaker notes

In your research, you specify a null hypothesis (typically labeled H0) and an alternative hypothesis (typically labeled Ha, or sometimes H1). By tradition, the null hypothesis corresponds to no change. When you are using Statistics to decide between these two hypothesis, you have to allow for the possibility of error. Actually, if you are using any other procedure, you should still allow for the possibility of error, but we statisticians are the only ones honest enough to admit this.

A Type I error is rejecting the null hypothesis when the null hypothesis is true.

Consider a new drug that we will put on the market if we can show that it is better than a placebo. In this context, H0 would represent the hypothesis that the average improvement (or perhaps the probability of improvement) among all patients taking the new drug is equal to the average improvement (probability of improvement) among all patients taking the placebo. A Type I error would be allowing an ineffective drug onto the market.

Remember that the hypotheses involve population parameters. Population parameters are impossible to compute. So you can only talk about Type I errors in an abstract sense. You will never know for certain if you have made a Type I error.

Alpha is the probability of a Type I error, and alpha is a value that you can compute. In most studies, researchers work hard to keep the probability of a Type I error low, typically at 5%.

What is a Type II error?

A Type II error is accepting the null hypothesis when the null hypothesis is false.
- False negative result
- Usually computed at MCD
- An example involving drug approval: a Type II error is keeping an effective drug off of the market.
\(\beta\) = P[Type II error]
Power = \(1-\beta\)

Speaker notes

A Type II error is accepting the null hypothesis when the null hypothesis is false. You should always remember that it is impossible to prove a negative. Some statisticians will emphasize this fact by using the phrase “fail to reject the null hypothesis” in place of “accept the null hypothesis.” The former phrase always strikes me as semantic overkill.

Many studies have small sample sizes that make it difficult to reject the null hypothesis, even when there is a big change in the data. In these situations, a Type II error might be a possible explanation for the negative study results.

Consider a new drug that we will put on the market if we can show that it is better than a placebo. In this context, H0 would represent the hypothesis that the average improvement (or perhaps the probability of improvement) among all patients taking the new drug is equal to the average improvement (probability of improvement) among all patients taking the placebo. A Type II error would be keeping an effective drug off the market.

It bears repeating that population parameters are impossible to compute. So you will never know for certain if you have made a Type I error.

Beta is the probability of a Type II error. Beta is a known quantity. Typically researchers try to keep beta small. 10% is a typical value, though in some settings, a Type II error rate as large as 20% could be tolerated.

Power is defined as 1-beta. I will talk more about power in a little bit.

What is a p-value?

Let t =
- (\(\bar{X}_1-\bar{X}_2\)) / se, or
- (\(\hat{p}_1-\hat{p}_2\)) / se, or
- r / se
p-value = Prob of sample result, t, or a result more extreme,
- assuming the null hypothesis is true
Small p-value, reject \(H_0\)
Large p-value, accept \(H_0\)

Alternate interpretations

Consistency between the data and the null
- Small value, inconsistent
- Large value, consistent
Evidence against the null
- Small, lots of evidence against the null
- Large, little evidence against the null

Speaker notes

There are two interpretations that I feel are more practical. You can think of the p-value as a measure of consistency between the data and the null hypothesis. A small value implies inconsistency. It is very unlikely that you will get a value like you’ve seen in your sample or a value more extreme under the assumption that the null hypothesis is true. So you should reject that assumption.

On the other hand if the sample results or anything more extreme has a high probability under the assumption that the null hypothesis is true, then you should feel comfortable accepting that assumption.

I have argued that the p-value is a measure of evidence. Some have called it a poor measure of evidence, but I stand by my interpretation.

If the p-value is small, you have lots of evidence against the null hypothesis. If the p-value is large, you have little or no evidence against the null hypothesis.

What the p-value is not (1/2)

A p-value is NOT the probability that the null hypothesis is true.
- P[t or more extreme | null] is different than
- P[null | t or more extreme]
  - P[null] is nonsensical
  - \(\mu\), \(\pi\), or \(\rho\) are unknown constants (no sampling error)

Speaker notes

The p-value is a conditional probability, and you always need to be careful about conditional probabilities. It is a probability about a sample result given an assumption about the population result. It is not a probability about a population result given the sample result. There are two reasons for this.

First, you can’t reorder a conditional probability. The probability of A given B is almost never the same as the probability of B given A. The example I give for this is the probability of being happy given that you are rich. That’s a pretty high number, I hope you’ll agree. There are a few rich people who lead miserable lives, but from everything I’ve seen, most rich people are pretty darn happy. The reverse of this is the probability of being rich given that you are happy. That number is much smaller. Because although I believe that money can buy happiness, a lot of other things can also buy happiness just as well. It’s not quite as easy to find happiness if you’re poor, but somehow, a lot of poor people find a way to be happy anyway.

A second reason that you can’t reverse the order is that you cannot make a probability statement about population parameters. They are numbers computed from the entire population, and are fixed values. You cannot make a probability statement about something that has no sampling error.

Only numbers computed from a sample (i.e., statistics) have sampling error.

What the p-value is not (2/2)

Not a measure FOR either hypothesis
- Little evidence against the null \(\ne\) lots of evidence for the null
Not very informative if it is large
- Need a power calculation, or
- Narrow confidence interval
Not very helpful for huge data sets

Speaker notes

The p-value is not a measure for either hypothesis. It is always a measure against a particular hypothesis. Now when the p-value is small, you can make a strong statement. We have lots of evidence against the null hypothesis. That translates into lots of evidence in favor of the alternative hypothesis.

When the p-value is large, however, you are in a quandary. Little or no evidence against the null hypothesis is not the same as lots of evidence for the null hypothesis.

It’s possible to have little or no evidence against the null and also have little or no evidence against the alternative. This happens whenever you have a really small sample size combined with a lot of noise.

You can’t prove a negative, so the saying goes. Well, you can prove a negative, but you have to work harder at it. A large p-value by itself is not persuasive, but if you combine it with a power calculation done prior to data collection, that’s pretty good evidence in support of the null hypothesis.

You could also combine a large p-value with a narrow confidence interval to support the null hypothesis. I’ll talk about that more in just a bit.

In general, the p-value is not very helpful for large samples. We’re seeing this more and more. Just about everything pops up as statistically significant with these huge data sets, and you can’t use the p-value to separate the important stuff from the trivial stuff. You need to look instead at the magnitude of the sample estimates and calculate how much uncertainty you can remove in your future predictions.

A research paper computes a p-value of 0.45. How would you interpret this p-value?

Strong evidence for the null
Strong evidence for the alternative
Little or no evidence for the null
Little or no evidence for the alternative
More than one answer above is correct.
I do not know the answer.

Figure 1: xkcd cartoon about jelly beans and cancer

Speaker notes

This cartoon is impossible to read, but you can find it on the Canvas site or in the readings. Here’s a brief run down.

In the first panel, a woman runs up to a man and shouts: Jelly beans cause acne!

The man replies : Scientists! Investigate!

In the second panel, one scientist, holding a clipboard announces: We found no link between jelly beans and acne (p > 0.05).

In the third panel, the woman says: I hear it’s only a certain color that causes it.

In a bunch of small panels, the scientist with a clipboard reports: We found no link between purple jelly beans and acne (p > 0.05).

We found no link between brown jelly beans and acne (p > 0.05).

We found no link between pink jelly beans and acne (p > 0.05).

The same for blue, teal, salmon, red, and so forth. And then…

We found a link between green jelly beans and acne (p < 0.05). An off-screen voice goes: Whoa!

The next six panels show

We found no link between mauve jelly beans and acne (p > 0.05).

We found no link between beige jelly beans and acne (p > 0.05).

We found no link between lilac jelly beans and acne (p > 0.05).

We found no link between black jelly beans and acne (p > 0.05).

We found no link between peach jelly beans and acne (p > 0.05).

We found no link between orange jelly beans and acne (p > 0.05).

At the bottom is a newspaper with the headline: Green Jelly Beans Linked To Acne! 95% Confidence. Only 5% chance of coincidence!

If you are interested in a transcript and a detailed explanation, https://www.explainxkcd.com/wiki/index.php/882:_Significant

What is p-hacking?

Abuse of the hypothesis testing framework.
- Run multiple tests on the same outcome
- Test multiple outcome measures
- Remove outliers and retest
Defenses against p-hacking
- Bonferroni
- Primary versus secondary
- Published protocol

Speaker notes

This is an example of p-hacking. You change the testing process to increase the probability of a Type I error (Rejecting the null hypothesis when the null hypothesis is true). This increases the chance of getting a positive result, which you may find desirable, but only by increasing the probability of a false positive result.

Some examples of p-hacking. Run multiple tests on the same outcome measure. Start with the regular t-test, include the t-test that allows for unequal variances, and run two different non-parametric tests, the Wilcoxon-Mann-Whitney test and the sign test. Choose the test with the smallest p-value.

You also might consider multiple outcome measures. Compare the mortality rate, the relapse rate, and the re-hospitalization rate. If any of the three is statistically significant, claim victory.

You could also do this with longitudinal data. Compare pain relief at one hour and at four hours. If you see a difference at one hour, claim that your new medication is faster acting. If you see a difference at four hours, claim that your medication is longer lasting.

You might run a test with the full data set and then with an outlier or two removed. Report for the data set that has the smaller p-value and pretend that this was your original choice all along.

These are only a few of the choices. I don’t want to say more because I feel like I’m the devil tempting you.

There are two defenses against p-hacking. Well three if you count being honest. But what I mean is there are two things that you can do that will satisfy others that you are playing fairly.

First, you can adjust your decision rule by using a Bonferroni correction. Bonferroni divides alpha by the number of tests. If you are using three different outcome measures, compare your p-value of 0.0133 instead of 0.05.

Second, you can designate one of your outcome measures as primary. If you achieve statistical significance on your primary outcome, great. The remaining outcome measures are secondary. If you achieve statistical significance on a secondary outcome meausure only, report the results as provisional and requiring independent replication.

You should publish a detailed protocol, either through a clinical trial registry, or now there are journals which accept publications of the research protocols before any data are collected. It’s a paper with literature review and methods section, but no results and no discussion section.

Now p-hacking has happened because some people have a skewed view of research. They are interested in using research to promote their own agenda rather than using research to uncover the truth. Perfectly understandable if you are a drug company, but you as an independent researcher should never try to skew the data. It hurts you and it hurts your patients. You need to adopt a disinterested posture in that you are glad when the research points in one direction and you are glad when it points in the opposite direction, because either way, you know more than you did before and you can treat your patients better because of this knowledge.

Break #5

What you have learned
- Tests of hypothesis
What’s coming next
- Confidence intervals

What is a confidence interval?

Range of plausible values
- Tries to quantify uncertainty associated with the sampling process.

Speaker notes

We statisticians have a habit of hedging our bets. We always insert qualifiers into our reports, warn about all sorts of assumptions, and never admit to anything more extreme than probable. There’s a famous saying: “Statistics means never having to say you’re certain.”

We qualify our statements, of course, because we are always dealing with imperfect information. In particular, we are often asked to make statements about a population (a large group of subjects) using information from a sample (a small, but carefully selected subset of this population). No matter how carefully this sample is selected to be a fair and unbiased representation of the population, relying on information from a sample will always lead to some level of uncertainty.

A confidence interval is a range of values that tries to quantify uncertainty associated with the sampling process.

Consider it as a range of plausible values.

There is a confidence level associated with any confidence interval, usually 95%, but sometimes 90% or 99%.

The confidence level is related to the alpha level (probability of a Type I error).

It also has a long range sampling interpretation.

If you repeatedly sampled from the same population, then 95% (or 90% or 99%) of the confidence intervals produced would contain the true value in the population.

Example of a confidence interval

Homeopathic treatment of swelling after oral surgery
- 95% CI: -5.5 to 7.5 mm
- Lokken P, Straumsheim PA, Tveiten D, Skjelbred P, Borchgrevink CF. Effect of homoeopathy on pain and other events after acute trauma: placebo controlled trial with bilateral oral surgery BMJ. 1995;310(6992):1439-1442.

Speaker notes

http://www.bmj.com/content/310/6992/1439.full

Always look for confidence intervals that are wide enough to drive a truck through. They are very good indicators of small sample sizes.

Consider a recent study of homoeopathic treatment of pain and swelling after oral surgery (Lokken 1995). When examining swelling 3 days after the operation, they showed that homoeopathy led to 1 mm less swelling on average. The 95% confidence interval, however, ranged from -5.5 to 7.5 mm. From what little I know about oral surgery, this appears to be a very wide interval. This interval implies that neither a large improvement due to homoeopathy nor a large decrement could be ruled out.

Now, you can’t drive a truck through an interval that goes from -5.5 to 7.5 mm, but from the perspective of a human mouth, this interval is huge. Generally when a confidence interval is very wide like this one, it is an indication of an inadequate sample size, an issue that the authors mention in the discussion section of this paper.

Confidence interval interpretation (1 of 7)

Figure 2: Interval that contains the null value

Confidence interval interpretation (2 of 7)

Figure 3: Interval entirely above the null value

Confidence interval interpretation (3 of 7)

Figure 4: Interval entirely below the null value

Confidence interval interpretation (4 of 7)

Figure 5: Interval entirely inside the range of clinical indifference

Speaker notes

You should also see whether the confidence interval lies partly or entirely within a range of clinical indifference. Clinical indifference represents values of such a trivial size that you would not want to change your current practice. For example, you would not recommend a special diet that showed a one year weight loss of only five pounds. You would not order a diagnostic test that had a predictive value of less than 50%.

Clinical indifference is a medical judgment, and not a statistical judgment. It depends on your knowledge of the range of possible treatments, their costs, and their side effects. As statistician, I can only speculate on what a range of clinical indifference is. I do want to emphasize, however, that if a confidence interval is contained entirely within your range of clinical indifference, then you have clear and convincing evidence to keep doing things the same way.

Confidence interval interpretation (5 of 7)

Figure 6: Interval partly inside/outside range of clinical indifference

Speaker notes

One the other hand, if part of the confidence interval lies outside the range of clinical indifference, then you should consider the possibility that the sample size is too small.

The interval contains zero, so it is plausible to behave as if the difference in population means or proportions is zero. But the interval also contains values that are clinically important. So it is plausible to behave as if there is a clinically important difference in means. How can you have two such different interpretations being plausible at the same time? That’s the definition of ambiguity. If you don’t like it, get used to it. Statistics will often identify areas of ambiguity, which is a good thing, because it tells us to not act prematurely, but instead demand more data before you make a definitive decision.

Quiz question, revisited

A research paper computes a confidence interval for a relative risk of 0.82 to 3.94. This confidence interval tells that the result is

statistically significant and clinically important.
not statistically significant, but is clinically important.
statistically significant, but not clinically important.
not statistically significant, and not clinically important.
The result is ambiguous.
I do not know the answer.

Speaker notes

Let’s go back to that question I posed earlier.

A research paper computes a confidence interval for a relative risk of 0.82 to 3.94. What does this confidence interval tell you.

Well, it tells you that a relative risk of 1 (equal risks) is plausible, but that a relative risk of 2 (a doubling of risk) is also plausible. A tripling of risk is plausible. Good grief! This is an ambiguous result.

Doesn’t this bother you? It should. Someone ran a terrible experiment. An experiment so poorly designed that it can’t distinguish between no change in risk, or a tripling of risk.

It’s a terrible thing, but it happens all the time and it doesn’t seem to bother anyone but me. This is wretched. You got a hundred patients to let you poke and prod them. They took some bitter pills or maybe placebos. They are sacrificing their bodies in the name of science. And the best you can do is a confidence interval that goes from 0.82 to 3.94. Hang your head in shame!

Confidence interval interpretation (6 of 7)

Figure 7: Confidence interval entirely inside the range of clinical indifference

Confidence interval interpretation (7 of 7)

Figure 8: Confidence interval entirely outside the range of clinical indifference

Speaker notes

Finally, if your confidence interval excludes the null value and lies outside the range of clinical indifference, then you have both statistical and practical significance.

Let’s talk about one more case. I don’t have a picture, but imagine a confidence interval that is mostly in the white region, the region of clinical importance, but the lower limit stretches into the range of clinical indifference. It doesn’t quite include the null value, but it comes within kissing distance. That’s a result that achieves statistical significance, but it does not provide definitive proof of clinical importance. No one ever talks about this case, but they should. Your confidence interval indicates statistical significance, but just barely. So don’t pretend that your results are the final word. You should not stop researching until you get a confidence interval that lies entirely inside or entirely outside the range of clinical indifference.

Why you might prefer a confidence interval

Provides same information as p-value,
- Clinical importance
- Distinguish between
  - definitive negative result, or
  - more research is needed

Break #6

What you have learned
- Confidence intervals
What’s coming next
- A simple R program

Grading rubric

Summary

What you have learned
- About this class
- R and RStudio
- Programming assignments
- Scales of measurement
- Tests of hypothesis
- Confidence intervals
- A simple R program